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1 . 0  INTRODUCTION 


Many  federal  agencies  are  investing  an  increasing  amount  of  money  in  the 
development  of  artificial  intelligence  (AI)  and  expert  systems  technology. 

This  reflects,  in  part,  a  desire  to  reduce  the  manpower- intensive  nature  of 
many  tasks  by  substituting  machine  intelligence  (and  energy)  where  possible, 
such  as  in  highly  structured,  fast-paced,  repetitive,  or  dangerous  situations. 
In  part,  however,  it  reflects  a  need  to  provide  assistance  to  decision-makers 
in  tasks  that  are  relatively  unstructured,  and  not  necessarily  repetitive, 
fast-paced  or  dangerous,  but  in  which  people's  information  processing 
capabilities  are  overtaxed.  This  report  deals  with  the  latter  case,  i.e.,  the 
use  of  AI  as  an  aid  to  decision  making. 

o 

AI  as  a  decision  aid  is  particularly  relevant  to  the  command  and  control  (C  ) 
and  management  settings,  because  those  are  settings  in  which  humans  are 
charged  with  both  authority  and  responsibility,  and  are  therefore  reluctant  to 
give  up  their  decision-making  role.  Whether  this  role  is  one  of  simply 
monitoring  and  overriding  (if  desired)  a  machine- initiated  decision,  or  of 
coming  to  a  decision  cooperatively  with  the  aid  of  machine  intelligence,  will 
depend  on  the  situational  characteristics,  but  in  either  case  the  expert 
system  supplements  rather  than  supplants  the  human,  in  the  case  of  interest 
here . 

Currently,  the  cutting  edge  of  research  on  expert  system  decision  aids  is 
concerned  with  self-learning  systems,  namely,  systems  that  can  make 
significant  changes  in  their  knowledge  base  and  their  internal  processing 
logic,  or  rules  of  behavior,  in  response  either  to  commands  of  the  user  or  to 
external  inputs  or  demands  that  have  been  placed  on  them.  It  is  tempting  to 
think  that  such  systems  will  be  significantly  more  flexible  than  conventional 
non-adaptive  systems  simply  because  they  are  potentially  applicable  to  a  more 
diverse  set  of  situations.  However,  the  potential  value  of  self -adaption  must 
be  assessed  against  the  cost  of  having  systems  that,  from  a  user's 
perspective,  are  constantly  changing  and  therefore  difficult  to  comprehend. 
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In  this  regard,  three  problem  areas  emerge: 

(1)  AI  decision  aids  in  a  C  context  deal  with  cognitive  rather  than 
perceptual  or  motor  functions;  cognitive  processes  are  generally  understood  to 
involve  the  structure  and  representation  of  information  or  knowledge,  and  the 
rules  or  logic  by  which  this  information  is  processed  to  provide  a  basis  for 
decisions  in  new  situations.  These  processes  cannot  be  directly  observed  in 
humans,  and  in  fact  the  difficulty  of  eliciting  the  knowledge  structures  and 
processing  rules  of  experts  is  one  of  the  chief  impediments  to  the  design  of 
expert  systems.  But  an  equally  difficult  problem  is  that  of  determining  what 
knowledge  and  logic  inherent  in  an  expert  system  is  needed  by  a  user  whose  way 
of  thinking  about  the  problem  may  not  match  that  of  the  system.  In  a  sense 
this  is  the  opposite  of  the  expert  knowledge  elicitation  problem,  since  in 
this  case  the  system's  knowledge  must  be  elicited  by,  or  at  least  made 
available  to,  the  system's  user. 

(2)  In  an  adaptive  or  self -learning  system  the  problem  is  compounded  by  the 
dynamic  nature  of  the  system's  expertise.  Thus,  new  information  that  has  been 
incorporated  into  the  system's  knowledge  structure,  and  new  or  modified  rules 
for  processing  the  information,  may  have  to  be  brought  to  the  attention  of  the 
user  rapidly,  when  appropriate,  in  order  to  up-date  the  user's  understanding 
of  the  system  and  maintain  his  confidence  in  its  outputs.  Except  for  some 
research  (to  he  discussed  later)  on  the  differences  between  experts  and 
novices  in  how  they  form  mental  models,  there  is  very  little  understanding  of 
the  evolution  of  knowledge  structures  or,  more  importantly,  what  significant 
changes  must  be  brought  to  the  attention  of  the  user  of  a  self- learning  expert 
system. 

(3)  Underlying  both  of  the  above  issues  is  the  requirement  for  an  effective 
(or  "friendly")  interface  between  the  system  and  its  user.  An  effective 
man/machine  interface  involves  more  than  simply  an  easily  understood  method  of 
interacting  (although  it  certainly  requires  that).  Cooperative  problem 
solving  between  an  expert  system  and  its  user  requires  the  ability  to  rapidly 
request  and  obtain  information,  share  knowledge  and  provide  prompts,  at  the 
initiation  of  either  the  system  or  its  user.  Furthermore,  if  the  user's 
confidence  in  the  system  is  to  be  maintained,  the  system  must  operate  in  a  way 
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that  is  predictable  by  its  user.  This  is  not  to  say  that  the  expert  system’s 
recommendations  must  be  anticipated,  but  rather  that  the  system’s  problem 
solving  procedures  must  be  predictable  so  that  the  user  can  interact  easily 
(i.e.,  be  able  to  interrupt  with  requests  or  overrides,  be  prepared  for  system 
requests)  in  order  to  achieve  a  truly  cooperative  approach  to  problem  solving. 
Methods  of  implementing  these  requirements  have  barely  been  covered  in  the 
literature  on  human/machine  interaction. 

In  summary,  in  order  for  self- learning  systems  to  achieve  their  full  potential 
as  decision  aids,  ways  must  be  found  to  determine  the  information  needs  of 
their  users  and  to  provide  their  users  with  dynamic,  on-line  up-dating  of  a 
changing  knowledge  base  and  logic,  and  with  interactive  techniques  that  ensure 
effective  cooperative  problem  solving. 

As  a  promising  approach  to  this  problem,  this  research  seeks  advances  in  both 
the  theory  and  application  of  mental  models  and  human  knowledge 
representation.  Regardless  of  how  knowledge  and  processing  rules  are 
represented  within  AI  systems,  their  understanding  by  a  user  will  depend  on 
the  ease  with  which  they  can  be  absorbed  into  the  user's  cognitive  framework. 
The  research  reported  here  explores  how  this  may  be  accomplished. 


-3- 


2.0  HUMAN  KNOWLEDGE  REPRESENTATION 


Research  on  how  humans  organize  or  represent  knowledge  has  been  prompted  by  a 
variety  of  motives,  all  of  which  relate  to  the  ultimate  improvement  of  human 
reasoning  and  problem-solving  performance.  Some  of  the  research,  described  in 
Kahneman,  Slovic  and  Tversky  (1982),  has  investigated  typical  heuristics  or 
short-cuts  in  the  reasoning  process  that  lead  to  biases  or  errors  in  judgment; 
this  work  is  motivated  largely  by  an  interest  in  helping  people  to  avoid 
violations  of  rational  or  prescriptive  models,  either  through  training  or 
computer-based  aids.  Some  developers  of  artificial  intelligence  systems  have 
sought  to  design  those  systems  to  emulate  human  information  processing 
structures,  in  order  to  achieve  the  speed  and  efficiency  shown  by  humans  under 
certain  conditions  (e.g.,  the  parallel  processing  capability  of  neural 
networks) .  Still  another  line  of  research  has  investigated  differences 
between  the  cognitive  structures  and  processes  of  experts  as  compared  with 
novices,  in  order  to  find  ways  to  design  instructional  systems  that  can  bring 
novices  to  expert  levels  more  efficiently.  The  several  lines  of  research  have 
produced  a  variety  of  categorization  schemes  that  reflect  different  theories 
about  how  people  represent  or  form  mental  models  of  knowledge.  These  schemes 
are  described  and  interpreted  in  this  section. 

2 . 1  Declarative  vs.  Procedural  Structures 

One  common  scheme  for  categorizing  knowledge  structures,  described  by  Madni 
(1988),  is  that  of  declarative  and  procedural  representations.  Declarative 
structures  refer  to  the  representation  of  facts  ("data")  or  assertions 
(logical  relations);  they  are  often  further  subdivided  into  relational 
structures  such  as  semantic  networks,  and  logical  structures  such  as 
production  rules,  frames  and  scripts,  although  production  rules  have  also  been 
viewed  as  procedural.  Procedural  representation,  on  the  other  hand,  can  be 
characterized  as  a  prescription  for  a  set  of  actions;  procedural  knowledge 
consists  of  knowledge  about  what  to  do  in  specified  situations.  Anderson 
(1982)  has  viewed  the  achievement  of  expertise  as  the  replacement  of  a 
declarative  mode  of  knowledge  representation  by  a  procedural  mode.  In  this 
view,  novice-  build  up  a  declarative  or  factual  base  of  knowledge,  and  learn  a 
set  of  analytical  tools  for  operating  on  it.  Their  problem  solving  tends  to 
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be  relatively  elaborate,  laborious  and  time-consuming,  the  level  of 
understanding  is  superficial  rather  than  deep,  and  their  ability  to  generalize 
to  other  situations  is  limited.  Declarative  knowledge  has  also  been  termed 
"propositional,"  by  Rumelhart  and  Norman  (1985),  among  others,  who  describe  it 
as  a  set  of  discrete  symbols,  propositions,  or  formal  statements.  As 
expertise  develops,  the  factual  knowledge  becomes  transformed  into  a  set  of 
procedures  that  can  be  brought  to  bear  quickly  and  intuitively  in  a  variety  of 
situations  (Madni,  1988).  Such  knowledge  is  characterized  by  Rumelhart  and 
Norman  as  knowledge  of  "how"  rather  than  "what,"  and  they  include  production 
rules  in  this  category. 

Using  this  categorization  scheme,  the  problem  solving  of  novices  is  seen  as 
the  application  of  analytical  methods  to  a  body  of  facts  and  relationships, 
while  that  of  experts  tends  to  be  a  more  intuitive,  relatively  automatic, 
invoking  of  a  set  of  procedures  appropriate  to  situations  with  common 
characteristics.  In  research  comparing  analytical  with  intuitive  styles  of 
decision  making,  Peters,  Hammond  and  Summers  (1974)  found  that  although 
analysis  results  in  greater  accuracy  on  the  average,  extreme  errors  can  occur 
with  a  significant  probability,  while  intuitive  thinking  results  in  a  flatter 
distribution  of  performance  (a  preponderance  of  approximately  correct 
responses)  but  with  relatively  few  extreme  errors. 

2 . 2  Interpretative  vs.  Generative  Structures 

Another  theoretical  approach  views  knowledge  structures  as  either 
interpretative  ("top-down")  or  generative  ("bottom-up").  Perhaps  the  most 
dominant  of  the  interpretative  theories  is  that  of  Schank,  Abelson  and  their 
colleagues  (Cullingford,  1978;  Schank,  1982;  Schank  and  Abelson,  1977)  based 
on  the  concept  of  schemata  or  frames.  They  posit  a  hierarchical  knowledge 
structure  that  provides  for  the  interrelation  and  relative  importance  of 
pieces  of  knowledge,  with  slots  at  each  node  in  the  hierarchy  corresponding  to 
pieces  of  knowledge  expected  to  fill  that  slot.  The  slots  serve  as  sources  of 
expectancies  as  to  what  information  will  be  encountered.  The  hierarchical 
organization  implies  a  top-down  manner  of  processing  and  representing 
information,  i.e.,  focus  will  be  on  the  higher,  more  important  goal -directing 
elements  which  in  turn  will  determine  the  organization  of  the  lower,  less 
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important  elements.  Schank  (1982)  has  emphasized  the  goal-directed  character 
of  memory  organization  in  his  concept  of  Memory  Organization  Packets  (MOPs), 
which  he  describes  as  a  set  of  scenes  directed  toward  the  achievement  of  a 
goal.  A  scene  common  to  several  situations  serves  as  an  index  from  one 
situation  to  another,  thereby  fostering  generalizations. 

On  the  other  hand,  generative  structures  are  characterized  by  knowledge 
organization  driven  by  lower  level  facts,  rules  or  relationships,  such  as 
production  rules,  or  semantic  networks.  It  seems  reasonably  clear  that  the 
data-driven  character  of  generative  structures  has  much  in  common  with  the 
declarative/analytic  representation  attributed  to  novices,  while  the 
goal-directed  character  of  interpretative  structures  is  similar  to  the 
procedural/intuitive  representation  attributed  to  experts. 

2 . 3  Analogical  Mental  Models 

The  literature  on  mental  models  significantly  overlaps  that  on  knowledge 
representation,  and  the  two  concepts  are  often  referred  to  interchangeably. 
However,  two  distinctions  that  have  been  drawn  are  that  (1)  a  mental  model 
bears  a  close  structural  isomorphism  (e.g.,  spatial,  temporal)  with  the 
phenomenon  or  system  it  represents  (Johnson-Laird,  1983),  and  (2)  it  should  be 
"runnable"  in  the  sense  that  a  person  should  be  able  to  modify  the  inputs  or 
relationships  mentally  and  determine  how  the  outputs  would  be  affected 
(Williams,  Hollan  and  Stevens,  1983).  Another  feature  commonly  ascribed  to 
mental  models,  although  not  in  distinction  to  knowledge  structures,  is  that 
their  components  may  be  decomposed  to  more  detailed  levels  or  aggregated  to 
broader  levels  of  abstraction. 

The  concept  of  a  runnable  mental  model  that  can  be  used  to  diagnose  reasons 
for  a  current  state  of  affairs  or  to  predict  future  conditions,  bears  a  close 
resemblance  to  the  type  of  knowledge  structure  described  above  as  declarative, 
analytical,  or  generative.  However,  it  is  generally  acknowledged  that  as  one 
becomes  familiar  with  a  class  of  problems  (i.e.,  as  one  approaches  expertise) 
one  can  use  a  briefer  version  of  the  model  in  which  a  set  of  conditions 
immediately  implies  the  appropriate  outputs,  thus  appearing  to  behave  much 
like  an  intuitive  knowledge  structure.  In  fact,  Klein  (1989)  has  emphasized 
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Che  role  of  analogical  or  recognition-based  reasoning,  thac  allows  experienced 
problem- solvers  to  recognize  key  aspects  of  a  situation  and  respond  very 
rapidly,  in  contrast  to  the  more  rigidly  formal  analytical  processes 
underlying  many  types  of  decision  aiding  systems.  Klein's  research  suggests 
that  analogical  decision  making  is  most  appropriate  when  what  is  wanted  is  a 
reasonably  satisfactory  rather  than  an  optimum  solution,  which  is  consistent 
with  the  findings  of  Peters,  Hammond  and  Summers  (1974)  described  earlier. 

2.4  Experts  vs.  Expert  Systems 

The  research  reviewed  above  suggests  that  when  experts  are  solving  a  problem 
manually,  their  knowledge  structure  or  model  of  the  problem  is  quite  different 
from  that  of  a  typical  expert  system  that  might  be  provided  as  a  decision  aid. 
The  expert's  view  of  the  problem  would  tend  to  be  analogous  to  the  problem 
domain,  reflecting  its  spatial  or  temporal  features,  and  his  or  her  approach 
would  tend  to  be  top-down  or  goal-driven  and  characterized  by  a  relatively 
simplified  set  of  heuristics  or  "intuitions,”  triggered  by  recognized  features 
of  the  situation  that  have  been  learned  over  a  long  period  of  time.  The 
expert's  knowledge  would  be  deep  enough  to  permit  rapid  generalization  to  new 
situations.  On  the  other  hand,  decision  aiding  systems  tend  to  be  data-driven 
and  therefore  data- intensive ,  relying  heavily  on  algorithmic  or  analytical 
procedures  that  seek  optimum  solutions.  This  discrepancy  in  knowledge 
structures  and  procedures  has  frequently  led  to  the  rejection  of  decision 
aiding  systems  by  experts,  and  is  responsible  for  a  substantial  current 
interest  in  finding  ways  to  design  such  systems  with  user  needs  in  mind. 

2 . 5  Research  Issues 

Although  it  appears  obvious  that  the  user's  mental  model  or  knowledge 
structure  should  be  a  driving  force  in  designing  interfaces  with  expert 
systems,  there  has  been  surprisingly  little  research  on  this  issue.  It  is  not 
clear,  for  example,  whether  interface  design  should  be  tailored  to  the  way  the 
expert  user  views  the  problem  (regardless  of  the  system's  knowledge 
structure) ,  or  whether  efforts  should  be  focused  on  ensuring  that  the  user  is 
given  an  accurate  model  of  the  structure  and  functioning  of  the  system  itself. 
One  study,  conducted  by  Lehner  and  Zirk  (1987),  examined  problem  solving 
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performance  by  a  human  and  an  expert  system  in  a  cooperative  mode. 

Cooperation  was  forced  by  the  fact  that  neither  party  alone  had  access  to  all 
the  data.  One  key  variable  was  the  consistency  of  the  problem  solving 
procedures  by  the  system  and  its  user.  The  other  key  variable  was  the 
accuracy  of  the  mental  model  that  the  human  had  of  the  system.  Under  one 
condition,  the  human  was  given  no  model  of  the  system;  in  the  "accurate 
cognitive  model"  condition  subjects  were  instructed  about  inference  networks 
in  general  and  were  shown  a  pictorial  display  of  an  inference  net  and  a  simple 
example  of  its  operation,  which  induced  a  cognitive  model  of  how  the  expert 
system  solved  problems.  The  results  showed  that,  regardless  of  the 
consistency  of  the  procedures  used,  performance  was  better  when  the  user  had  a 
good  cognitive  model  of  the  system's  procedures.  In  fact,  under  the  "accurate 
cognitive  model"  condition,  performance  was  better  when  the  user  and  the 
system  were  inconsistent  rather  than  consistent  in  their  procedures.  These 
results  suggest  that  expert  systems  and  humans  can  indeed  complement  each 
other  by  using  different  problem  solving  methods,  provided  that  the  human 
understands  the  basis  on  which  the  system  reaches  its  conclusions.  Implied  in 
these  results  is  the  importance  of  notifying  the  user  rapidly  of  any  changes 
in  the  system's  procedures  that  might  affect  the  cooperative  interaction. 
Lehner  and  Zirk  used  a  set  of  written  instructions  in  their  experiments,  but 
clearly  in  the  case  of  a  self- learning  system  the  information  would  have  to  be 
developed  internally  by  the  system  and  presented  by  means  of  an  appropriately 
designed  user  interface. 

One  of  the  limitations  of  Lehner  and  Zirk's  study  was  that  the  subjects  were 
not  experts  in  the  usual  sense  of  the  term.  The  task  was  a  relatively  simple 
one  that  could  be  learned  in  a  brief  training  session,  and  in  that  sense  the 
subjects  could  not  be  considered  novices.  However,  expertise  is  usually 
characterized  by  years  of  experience  dealing  with  many  variants  of  the  same 
type  of  problem,  and  the  development  of  a  relatively  stable  view  of  the 
problem  domain  and  set  of  preferred  procedures  and  heuristics  for  solving  it. 
These  were  not  characteristics  of  the  subjects  in  their  study. 

The  question  remains,  therefore,  as  to  the  most  appropriate  approach  to 
designing  interfaces  between  an  expert  user  and  an  expert  system,  namely, 
whether  to  base  the  design  on  the  user's  mental  model  of  the  problem  domain. 
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or  to  attempt  to  foster  a  good  mental  model  of  the  expert  system  itself. 
Indeed,  recent  research  by  Roth,  Bennett  and  Woods  (1987)  suggests  the 
importance  of  a  shared  frame  of  reference  for  the  user  and  the  machine  aid. 
These  authors  stress  the  importance  of  explicitly  representing  to  the  user  the 
machine's  perception  of  the  status  of  the  problem  it  is  attempting  to  solve. 
Moray  (1987),  on  the  other  hand,  argues  that  operators  of  systems  acquire 
mental  models  of  these  systems  that  are  "homomorphs"  of  the  original  system, 
by  which  he  means  reduced  or  simplified  versions  (partial  models)  as 
contrasted  with  isomorphs  or  exact  models.  He  advocates  that  intelligent 
decision  aids  intended  to  help  operators  control  systems  should  be  based  on 
these  homomorphs . 

The  research  reported  here  is  an  attempt  to  explore  these  issues  in  a  military 
decision  making  context.  The  problem  domain  is  that  of  the  prioritization  of 
targets  for  tactical  air  strikes,  during  which  a  simulated  expert  system 
develops  a  recommended  target  priority  list  and  the  expert  user's  task  is  tc 
determine  why  the  system  developed  the  list  that  it  did.  Specifically,  the 
research  explored  the  issues  of  (1)  whether  it  is  more  important  for  displays 
to  be  compatible  with  the  user's  or  the  aid's  model  of  the  problem,  and  (2) 
the  extent  to  which  the  displays  affect  performance  when  the  user  has  or  does 
not  have  a  good  mental  model  of  the  aid. 
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3.0  PROCEDURE 


3.1  general 

The  general  procedure  was  to  ask  Tactical  Air  Force  targeting  specialists  to 
work  on  a  targeting  problem,  except  that  instead  of  generating  a  target 
prioritization  list  they  were  to  examine  a  strike  plan  generated  by  a 
simulated  Target  Prioritization  Tool  (TPT) ,  and  answer  some  questions  designed 
to  measure  how  well  they  understood  why  the  TPT  generated  the  list  and  plan 
that  it  did.  All  the  participants  were  given  the  same  mission  guidance  (to 
achieve  air  superiority  in  the  area  of  interest  in  three  days)  and  the  same 
scenario  (enemy  air  order  of  battle).  They  differed  in  (1)  whether  they  were 
provided  with  a  relatively  good  or  relatively  poor  model  of  how  the  TPT 
worked,  and  (2)  whether  they  were  provided  with  available  displays  that  were 
relatively  user-oriented  or  relatively  aid-oriented.  Thus,  the  experimental 
design  consisted  of  four  cells,  as  shown  in  Figure  3-1. 


Good  aid  model 
User-oriented 
displays 

Good  aid  model 
Aid-oriented 
displays 

Poor  aid  model 
User-oriented 
displays 

Poor  aid  model 
Aid-oriented 
displays 

Figure  3-1:  Experimental  design. 

The  computations  leading  to  the  TPT  recommendations  were  based  on  the  types  of 
data  and  algorithms  utilized  in  a  Target  Prioritization  Aid  (TPA)  developed  by 
PAR  (in  conceptual  form)  for  Rome  Air  Development  Center  (reference).  All 
participants  were  briefed  on  how  the  TPT  worked.  Those  who  were  in  the  "Good 
aid  model"  groups  were  given,  in  addition,  a  one-page  summary  of  the 
step-by-step  procedure  followed  by  the  TPT  in  reaching  its  conclusions,  while 
those  in  the  "Poor  aid  model"  groups  were  not  given  this  summary.  Those  in 
the  "User-oriented  displays"  groups  were  given  displays  based  on  an  expert 
consultant's  description  of  how  targeteers  do  their  job;  these  displays 
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included  maps  of  the  area  of  interest  shoving  16  enemy  air  bases  (possible 
targets)  with  their  aircraft  (types  and  numbers),  and  more  detailed 
information  about  each  base  and  its  installations  (Air  Installation  Data)  in 
tabular  form,  base  by  base.  This  was  consistent  with  the  concept  of  an 
expert's  mental  model  of  the  problem  as  geographic,  top-down,  and  simplified. 
Participants  in  the  "Aid- oriented  displays"  groups  were  given  the  same 
information  in  matrix  form,  each  matrix  providing  data  on  all  enemy  bases  in 
the  same  table.  This  was  consistent  with  the  internal  structure  of  the  TPT  as 
a  data- intensive ,  algorithmic  optimizer.  The  TPT  performs  several  functions 
based  on  values  that  had  been  furnished  by  experts  during  the  development  of 
the  Target  Prioritization  Aid.  It  estimates  the  potential  sortie  generation 
rate  (the  threat)  for  each  enemy  airbase,  based  on  type  and  number  of  aircraft 
and  distance  from  the  area  of  interest.  It  estimates  the  reduction  in  sortie 
generation  rate  that  could  be  achieved  by  one  or  more  successful  attacks  on 
specific  airbase  components  such  as  unsheltered  aircraft,  revetments, 
maintenance  facilities,  etc.  (the  benefit).  It  estimates  the  number  of 
friendly  strike  aircraft  required  to  attack  those  components  (the  cost).  It 
calculates  the  benefit/cost  ratio  for  attacks  on  those  components,  aggregates 
these  values  across  components  for  each  airbase,  prioritizes  the  potential 
targets  according  to  these  values,  generates  a  prioritized  list  of  targets  and 
specifies  the  components  to  be  hit  at  each  base.  All  participants  were 
provided  with  the  "knowledge  base"  of  the  TPT,  that  is,  the  expert  assessments 
on  the  basis  of  which  the  threats,  benefits,  and  costs  were  estimated,  and  the 
aggregated  benefit/cost  ratio  for  each  target.  The  "Good  Aid  Model"  subjects 
were  given,  in  addition,  the  one-page  summary  of  the  aid's  processes,  as  shown 
in  Figure  3-2. 

The  TPT  was  simulated  in  storyboard  form.  The  displays  available  to 
participants  were  presented  as  hard-copy  pages  in  a  notebook,  with  the  pages 
tabbed  for  ease  of  retrieval.  Prior  to  each  experimental  trial,  the 
participant  was  "walked"  through  the  notebook  and  each  available  display  was 
explained.  • 

In  the  actual  experimental  trial,  the  participant  was  given  the  "output"  of 
the  TPT,  consisting  of  the  prioritized  target  list  and  components  to  be  hit  at 
each  base,  for  each  of  three  successive  days  of  attack.  The  three  highest 
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DESCRIPTION  OF  AID  PROCESSES 


The  TPT  determines  the  priority  of  targets  by  proceeding  through  the  following 

steps . 

1.  Calculate  #  Sorties  of  Interest  -  Calculate  the  number  of  sorties  of  in¬ 
terest  that  each  airbase  can  generate  per  day. 

2.  Calculate  Effects  -  For  each  component  of  each  airbase  calculate  (using 
the  database  of  Expert  Assessment  of  Effects)  the  percent  reduction  in 
air  base  capacity  that  would  be  achieved  by  attacking  that  component 
either  once  or  twice. 

3.  Calculate  #  Sorties  Reduced  -  Multiply  the  above  two  numbers  to  deter¬ 
mine  the  number  of  enemy  sorties  that  would  be  reduced  by  hitting  each 
component  of  each  airbase. 

4.  Calculate  Friendly  Sorties  Req'd  -  Calculate  the  number  of  friendly 
sorties  (F-16  and  F-llls)  required  to  attack  each  component  of  each  air¬ 
base. 

5.  Calculate  Component  Benefit-to-Cost  Ratios  -  For  each  component  of  each 

airbase  calculate  a  benefit- to-cost  ratio  by  dividing  the  number  of 
enemy  sorties  reduced  (step  3)  by  the  average  number  of  friendly  sorties 
required  (step  4).  A  separate  value  is  calculated  for  hitting  the  com¬ 
ponent  either  once  or  twice. 

6.  Determine  Plan  of  Attack  -  Select  for  attack  the  components  (of  any  air¬ 

base)  with  the  highest  benefit- to-cost  ratios.  Select  enough  components 
to  use  more  than  300  F-16  sorties  in  a  ground  attack  role.  This  is  the 
overall  proposed  Plan  of  Attack. 

7.  Calculate  Benefit-to-Cost  Ratio  for  each  airbase  -  For  each  airbase  0  or 
more  components  will  have  been  targeted  in  the  Plan  of  Attack.  For  each 
airbase,  calculate  the  total  benefit- to-cost  ratio  for  hitting  all  the 
components  targeted  in  the  Plan  of  Attack. 

This  benefit-to-cost  ratio  is  the  target  value  (priority)  assigned  to  an 
airbase. 


Figure  3-2.  Description  of  Aid  Processes. 
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priority  targets  on  each  day  were  identified  (they  were  not  necessarily  the 
sane  targets  each  day),  and  the  task  for  the  participant  was  to  select,  from  a 
list  of  possible  factors,  those  that  made  a  significant  contribution  to  the 
high  value  of  the  top  three  targets,  and  to  rank  the  components  hit  at  each 

target  (they  were  not  necessarily  the  same)  in  order  of  their  importance  to 

that  target's  high  value. 

Appendix  A  presents  the  instructions,  guidance,  and  response  forms  given  to 
all  the  subjects;  the  instructions  and  guidance  were  read  to  them  as  well  as 

being  provided  in  written  form.  Appendix  B  presents  the  displays  made 

available  to  the  User-Oriented  Displays  Group.  Appendix  C  presents  the 
displays  made  available  to  the  Aid-Oriented  Displays  Group. 

In  addition  to  the  performance  measures ,  a  tabulation  was  made  of  the 
frequency  with  which  each  display  was  referred  to.  Finally,  at  the  end  of  the 
session,  each  subject  was  asked  which  displays  were  found  particularly 
helpful,  what  other  displays  would  be  desirable,  and  how  in  general  a 
computerized  aid  could  help  in  the  targeting  task. 

A  pilot  study  was  conducted  at  Headquarters,  Tactical  Air  Command,  Langley  Air 
Force  Base  with  four  Targets  personnel,  each  of  whom  was  available  for  two 
2 -hour  sessions  on  two  successive  days.  It  had  originally  been  intended  to 
present  two  problems  in  each  session,  with  changed  values  in  the  expert 
knowledge  base  or  the  enemy  AOB,  to  determine  how  well  the  changes  could  be 
detected.  However,  the  task  proved  to  be  more  difficult  than  expected. 
Consequently,  the  procedure  was  simplified  by  presenting  only  one  problem  and 
relying  on  the  changes  in  the  TPT's  recommendations  over  the  three-day  strike 
planning  period  to  determine  how  well  the  TPT's  basis  for  target 
prioritization  could  be  understood.  As  a  result  of  the  pilot  study  certain 
changes  were  made  in  the  response  sheets  to  simplify  and  shorten  the  time 
needed  to  record  responses.  The  analysis  is  based  on  results  obtained  one 
week  later,  with  8  Targets  personnel  at  9th  Air  Force,  Shaw  Air  Force  Base. 
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3.2 


The  four  Targets  personnel  at  Langley  AFB  were  from  the  Directorate  of 
Intelligence  Application,  Targets  Division  (INAT);  they  included  two  Majors,  a 
Technical  Sergeant  and  a  Staff  Sergeant  whose  years  of  experience  doing 
targeting  ranged  from  0-15  years.  These  participants  contributed 
significantly  to  the  project  by: 


•  Providing  a  clearer  understanding  of  the  problem  domain  which  in 
turn  permitted  the  development  of  improved  displays  and  test 
methodology; 

•  Providing  the  expertise,  advice  and  guidance  that  allowed  us  to 
proceed  to  Shaw  AFB  with  a  proven  test  plan; 

•  Facilitating  the  arrangements  for  participants  at  Shaw  AFB. 


The  8  Targets  personnel  at  Shaw  AFB  were  from  the  9th  Air  Force's  Tactical 
Intelligence  Squadron,  Targets  Branch;  their  experience  doing  targeting  ranged 
from  2-12  years,  with  a  mean  of  4.8  years.  All  the  participants  at  both 
locations  were  extremely  cooperative. 
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4.0  RESULTS 


Described  below  are  the  results  for  the  eight  participants  from  Shaw  AFB.  The 
determination  that  the  Langley  experiment  would  be  treated  as  a  pilot  study 
was  made  prior  to  any  data  compilation  or  analysis. 

4 . 1  Performance  Analysis 

For  each  of  the  top  three  priority  targets,  for  each  day,  subjects  were 
required  to  (1)  select  from  the  List  of  Factors  those  factors  that  led  to  a 
target  receiving  a  high  priority  and  (2)  rank  order  the  target  components 
selected  for  attack  by  their  contribution  to  the  overall  target  value.  For 
both  of  these  assessments  correct  answers  were  derived  (see  Appendix  D) . 
Determining  the  "correct"  answer  involved  some  judgment  on  our  part.  However, 
the  assessment  of  the  "correct"  answer  was  made  prior  to  any  examination  of 
the  data. 

4.1.1  Selection  of  factors.  For  each  day,  each  factor  was  coded  as  either 
"Yes"  (clearly  a  factor) ,  "No"  (clearly  not  a  factor) ,  or  "Uncertain" 
(intermediate  range).  For  instance,  Factor  2  was  "Airbase  contains  relatively 
more  enemy  a/c  of  interest."  Each  airbase  had  either  45,  30,  15,  or  0 
aircraft  of  interest.  If  the  airbase  had  45,  Factor  2  was  labeled  "Yes;"  15 
or  0  it  was  labeled  "No;"  and  30  it  was  labeled  "Uncertain."  For  each  high 
priority  target  on  each  day  we  determined  the  number  of  "Yes"  or  "No"  factors 
correctly  identified.  For  instance,  for  airbase  Mimon  on  Day  1  there  are 
three  "Yes"  and  two  "No"  factors.  If  a  subject  listed  as  important  two  "Yes" 
factors  and  one  "No"  factor,  then  he  or  she  received  a  score  of  3  out  of  5. 
Note  that  mathematically.  Factor  5  is  derived  from  and  redundant  with  Factor 
4.  Consequently,  rather  than  double  count  the  same  factor,  it  was  dropped 
from  consideration  prior  to  any  data  analysis. 

After  recording  the  scores,  we  realized  that  Factor  7  was  (unintentionally) 
deceptive.  As  it  turns  out,  it  is  inherent  in  the  nature  of  the  TPA  algorithm 
that  Factor  7  must  always  be  answered  "No,"  no  matter  what  targets  are 
selected.  This  was  a  subtle  point,  and  nearly  all  the  subjects 
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inappropriately  added  Factor  7  to  their  list  of  factors.  We  performed  the 
analysis  both  with  and  without  this  factor. 

Data  for  each  subject,  and  totals,  are  found  in  Tables  4-1  through  4-5.  Table 
4-1  shows  the  total  number  of  factors  correct  by  cell.  Table  4-2  shows 
identical  data  to  Table  4-1,  except  that  Factor  7  was  dropped. 


Table  4-1:  Number  of  Factors  Correctly  Identified 

(Out  of  43) 


Conceptual 
Model 


Display 

User-Oriented  Aid-Oriented 


Good 

S  #1 

S  #8 

28 

11 

47 

S  #2 

S  #7 

27 

37 

64 

S  #4 

21 

S  #3 

33 

Poor 

S  #5 

25 

S  #6 

17 

46 

50 

Totals 

111 

96 


Totals 


93 


114 


Table  4-2:  Number  of  Factors  Correctly  Identified,  Excluding  Factor  7 

(Out  of  34) 

Display 


Conceptual 
Model 


User-Oriented 


Aid-Oriented 


Good 

S  #1 

S  #8 

28 

19 

47 

S  #2 

S  #7 

26 

32 

58 

S  #4 

21 

S  #3 

29 

Poor 

S  #5 

11 

S  #6 

17 _ 

40 

46 

Totals 

105 

86 


Totals 


87 


104 


With  only  two  subjects  per  cell,  and  a  wide  variation  between  subjects,  a 
by-subject  statistical  analysis  of  proportion  correct  will  not  result  in  any 
statistically  significant  results.  To  determine  whether  or  not  there  was  any 
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discernible  difference  between  the  Good  vs.  Poor  Conceptual  Model  conditions 
and  the  User-  vs.  Aid-Oriented  Display  conditions,  a  Chi-square  analysis  was 
performed  on  the  overall  number  correct  per  cell.  For  Good  vs.  Poor 
Conceptual  Model  the  p-levels  were  .35  >  p  >  .3  and  .25  >  p  >  .15  for  Tables 
4-1  and  4-2,  respectively;  thus  the  good  conceptual  model  resulted  in  slightly 
but  not  significantly  better  performance.  For  User-  vs.  Aid-Oriented  Displays 
the  p-levels  were  .2  >  p  >  .1  and  .25  >  p  .2  for  Tables  4-1  and  4-2, 
respectively;  with  the  aid-oriented  displays  resulting  in  slightly  but  not 
significantly  better  performance.  The  best  single  condition  was  Good 
Conceptual  Model  combined  with  Aid-Oriented  Displays. 

4.1.2  Prioritization  of  components.  To  determine  the  correct  priority  of  the 
components,  we  calculated  the  benefit/cost  ratio  for  each  component  (using 
information  available  to  the  subjects)  and  ordered  the  components  according  to 
this  ratio.  We  compared  the  subjects'  ranking  to  the  correct  ranking  by 
decomposing  each  ranking  into  separate  pair  comparisons  and  then  recording  the 
proportion  of  pair  comparisons  the  subjects  ordered  correctly.  This  procedure 
is  conceptually  similar  to  Kendall's  Tau  measure,  but  allows  us  to  aggregate 
the  data  from  multiple  rank  orders. 

After  scoring  the  subjects,  we  noticed  that  the  TPA  algorithm  invariably 
assigned  a  high  benefit/cost  ratio  to  the  component  "Unsheltered  Aircraft" 
whenever  the  Air  Installation  Data  indicated  there  were  only  one  or  two 
unsheltered  aircraft  at  the  air  base.  Consequently,  for  this  component  the 
TPA  algorithm  was  in  direct  contrast  to  any  reasonable  answer.  For  this 
reason  we  analyzed  the  data  with  and  without  this  component. 

Tables  4-3  and  4-4  present  the  data  by  cell.  The  Good  vs.  Poor  Conceptual 
Model  results  were  similar  to  the  selected- factors  data.  The  User-  vs. 
Aid-Oriented  Displays  results  were  reversed.  Subjects  with  user-oriented 
displays  performed  better.  Here  the  best  single  condition  was  Good  Conceptual 
Model  with  User-Oriented  Displays. 
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Table  4-3:  Number  of  Pairs  of  Components  Correctly  Ordered 

(Out  of  55) 


Conceptual 
Model 


Display 

User-Oriented  Aid-Oriented 


Good 

S  #1 

S  #8 

43 

22 

72 

S  #2 

S  #7 

26 

22 

58 

S  #4 

20 

S  #3 

31 

Poor 

S  #5 

22 

S  #6 

21 

55 

52 

Totals 


127 


110 


Totals 

130 

107 


Table  4-4:  Number  of  Pairs  of  Components  Correctly  Ordered, 

Excluding  Unsheltered  A/C 
(Out  of  35) 

Display 


Conceptual 
Model 


User-Oriented 


Totals 


87 


Aid-Oriented 


Good 

S  #1 

S  #8 

26 

21 

47 

S  #2 

S  #7 

22 

21 

43 

S  #4 

18 

S  #3 

21 

Poor 

S  #5 

22 

S  #6 

16 

40 

37 

80 


Totals 

90 

77 


As  above,  we  performed  a  Chi-square  analysis  on  the  total-number-correct  per 
condition.  For  the  Good  vs.  Poor  Conceptual  Model  difference  the  p- levels 
were  .2  >  p  >  .1  and  .4  >  p  >  .3  for  Tables  4-4  and  4-5,  respectively.  For 
the  User  vs.  Aid-Oriented  Displays  the  p- levels  were  .35  >  p  .25  and  p  >  .5 
for  Tables  4-3  and  4-4,  respectively. 


4.1.3  Summary  of  performance  analysis.  We  also  examined  the  correlation 
between  rank  and  years  of  experience  with  this  performance  data.  No 
correlation  between  performance  and  these  variables  was  discernible. 


Overall  the  only  reliable  effect  seems  to  be  for  the  Good  vs.  Poor  Conceptual 
Model  difference.  Performance  in  the  Good  Conceptual  Model  groups  were 
generally  higher  than  performance  in  the  corresponding  Poor  Conceptual  Model 
groups.  Note,  however,  that  although  the  Chi-square  tests  indicate  some 
marginally  significant  results,  the  observations  could  hardly  be  considered 
independent,  since  each  subject  contributed  multiple  data  points.  In  short, 
although  these  tests  do  indicate  a  possible  effect  for  the  Good  vs.  Poor 
Conceptual  Model  conditions,  the  analysis  does  not  allow  us  to  attribute  the 
effect  uniquely  to  the  experimental  manipulations. 

4.1.4  Pattern  of  use  of  displays.  Of  interest  is  whether  the  differing 
conditions  impacted  the  displays  which  the  subjects  examined  while  solving  the 
experimental  problems.  Of  particular  interest  is  the  extent  to  which  subjects 
used  the  Expert  Assessment  (Knowledge  Base)  displays  to  solve  the  problems. 

For  each  subject,  we  recorded  the  number  of  times  they  examined  each  display. 
Table  4-5  indicates,  for  each  subject,  the  number  of  times  a  subject  examined 
any  of  the  three  Expert  Assessment  displays  and  (in  parentheses)  the  total 
number  of  times  attention  was  shifted  from  one  display  to  another.  In 
general,  the  Aid-Oriented  Displays  group  examined  the  Expert  Assessment 
displays  far  more  frequently  than  the  User-Oriented  Displays  group.  A  t-test 
comparison  resulted  in  p-levels  of  .1  >  p  >  .05  (two-tailed)  for  both 
proportions  and  absolute  numbers. 


Table  4-5:  Number  of  Times  Any  of  the  Three  Expert  Assessment 

Displays  was  Examined 


Display 


Conceptual 

Model 


Good 


Poor 


User-Oriented 

Aid-Oriented 

S  #1  2  (105) 

S  #2  37  (159) 

S  #8  43  (195} 

S  #7  31  (2011 

45  (300) 

68  (360) 

S  #4  1  (109) 

S  #3  40  (183) 

S  #5  0  (157} 

S  #6  45  (148") 

1  (266) 

85  (331) 

Totals 


113 

(660) 


86 

(597) 


Totals  46  (566) 


153  (691) 
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Note  that  unlike  the  Chi-square  tests  above,  the  t-test  treated  each  subject 
as  a  single  data  point.  Consequently,  statistical  differences  between  the  two 
conditions  could  be  reasonably  attributed  to  the  experimental  manipulations. 
(Note  however  that  this  test  was  determined  a  posteriori.) 

4.1.5  Effect  of  display  use  on  performance.  Although  participants  with  the 
aid- oriented  displays  tended  to  use  the  Expert  Assessment  tables  more  than 
those  with  the  user-oriented  displays,  an  interesting  question  is  whether  the 
use  of  these  displays  affected  performance,  for  better  or  worse.  No 
statistical  analysis  was  justified  with  only  8  subjects,  but  an  examination  of 
the  scores  and  frequency  of  use  data  was  made.  It  turns  out  that  of  the  four 
subjects  who  referred  to  those  displays  most  frequently,  two  scored  highest 
and  two  scored  lowest  on  the  Identification  of  Critical  Factors,  and  there  was 
no  discernible  trend  in  the  Ordering  of  Components.  Thus,  although  several  of 
the  participants  commented  that  these  tables  were  useful,  there  is  no  evidence 
that  performance  was  significantly  improved  by  frequent  reference  to  them. 

4 . 2  Qualitative  Comments 

It  had  been  anticipated  that  those  subjects  in  the  User-Oriented  Display 
groups  would  make  frequent  reference  to  map  displays  available  to  them,  since 
currently  targeteers  rely  heavily  on  map  displays.  Contrary  to  expectation, 
the  map  displays  were  almost  never  referenced.  Most  of  the  participants 
recognized  the  value  of  tabular  data  for  solving  the  task  they  were  given, 
especially  since  the  matrices  allowed  them  to  compare  characteristics  of  all 
possible  targets  very  quickly.  Only  one  participant  expressed  a  preference 
for  an  "ideal"  set  of  displays  that  would  promote  a  top-down  approach:  a  map 
of  the  total  situation  (bases,  distances  and  sorties),  as  well  as  individual 
tables  with  details  that  could  be  selected  as  desired. 

Another  class  of  comments  suggested  disagreements  with  the  criteria  used  by 
the  TPT  to  prioritize  targets.  The  most  fundamental  was  the  objection  to  use 
of  sortie  reduction  rate  as  the  primary  criterion  (despite  the  Commander's 
Guidance  which  established  the  goal  of  air  superiority  in  three  days).  One 
subject  commented  that  friendly  forces  should  inflict  as  much  damage  as 
possible,  implying  that  a  procedure  based  on  aggregated  benefit/cost  ratios 
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would  not  be  the  accepted  procedure  in  the  field.  Other  participants 
apparently  did  not  appreciate  the  TPT's  selection  of  components  based  on 
effects  two  or  three  days  in  the  future;  they  would  have  hit  aircraft  in  the 
open  or  in  shelters  immediately,  ignoring  the  short-term  effects  of  that 
strategy  as  compared  with  the  longer-term  effects  of  hitting  other  components. 
The  value  of  hitting  command  and  control  facilities  immediately  was  justified 
by  reference  to  their  critical  importance  in  view  of  the  enemy's  hierarchical 
structure  and  the  immediate  (if  indirect)  effect  of  their  loss  to  enemy 
operational  capability. 

Such  comments  are  important  even  beyond  their  application  to  the  type  of 
expert  system  hypothesized  in  this  study.  They  suggest  that  users  of  expert 
systems  must  be  informed  not  only  of  numerical  values,  generated  by  experts, 
that  go  into  algorithmic  computations,  but  also  of  the  rationale  underlying 
these  judgments.  The  user  of  such  a  system  should  use  its  recommendations  as 
one  source  of  information- -all  the  participants  agreed  with  that  concept. 

They  should  be  able  to  disagree,  based  on  either  general  or  situational 
differences .  But  it  is  important  to  be  able  to  identify  the  source  of 
disagreement  quickly  in  order  to  take  effective  action,  and  to  achieve  this 
the  user  needs  more  information  about  the  expert  system  than  was  provided  in 
this  exercise. 
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5.0  DISCUSSION  OF  RESULTS 


The  results  of  this  research  are  only  suggestive.  The  task  presented  to  the 
subjects  was  relatively  difficult,  and  the  subjects,  although  currently 
assigned  to  an  appropriate  office,  did  not  have  as  much  experience  in 
targeting  as  we  had  hoped  (with  one  or  two  exceptions)  .  There  were  only  eight 
subjects,  and  they  differed  widely  in  experience.  Only  two  hours  were 
available  to  explain  the  problem,  describe  the  aid  and  displays,  present  the 
scenario,  and  work  the  problem.  With  so  much  material  to  be  absorbed  before 
starting  the  problem,  the  group  receiving  the  "good"  conceptual  model  of  the 
aid  (a  one-page  summary  of  its  step-by-step  procedures)  might  not  have  had 
much  advantage  over  the  "poor"  conceptual  model  group.  The  three  displays 
considered  most  important  for  understanding  the  aid's  reasoning  were  common  to 
both  the  Aid-Oriented  and  User-Oriented  Display  group. 

On  the  other  hand  the  participants  in  this  study  may  well  have  been  typical  of 
the  users  of  an  expert  targeting  system  and,  in  fact,  of  military  expert 
systems  generally.  Very  senior  personnel  with  broad  experience  would  probably 
have  moved  up  in  the  chain  of  command  to  relatively  more  supervisory 
positions,  while  the  expert  system  would  have  been  designed  as  an  aid  for 
middle- level  personnel  with  relatively  less  experience.  What  may  be  inferred 
from  the  results  of  this  experiment  is  that  such  a  group  of  expert  system 
users  are  likely  to  be  highly  variable  in  their  type  and  extent  of  operational 
experience,  and  that  therefore  the  system  must  be  designed  with  a  wide  range 
of  user  capabilities  in  mind.  In  fact,  for  many  users  the  system  should  serve 
the  dual  purposes  of  decision  aiding  and  training,  and  the  training  function 
should  be  aimed  not  only  at  teaching  users  how  the  system  works  but  also  at 
providing  them  with  an  opportunity  to  practice  solving  the  relevant  problem 
under  a  variety  of  situations. 

The  discussions  following  each  session  were  revealing  of  both  the  problems 
faced  by  the  participants  and  the  paths  toward  their  solution.  It  was  clear 
that  targeteers  have  their  own  ideas  of  the  target  characteristics  that 
contribute  to  high  priority.  These  may  vary  among  individuals,  and  they 
certainly  may  vary  from  those  of  experts  whose  views  have  been  solicited  as  a 


-22- 


basis  for  design  of  an  expert  system.  If  the  expert  system  data  or  model 
change  as  a  result  of  changes  in  the  situation,  or  if  its  recommended  plans 
over  a  period  of  time  are  based  on  time-varying  criteria  of  effectiveness,  the 
rationale  for  its  recommendations  must  be  clear  to  the  user  so  that 
differences  of  viewpoint  can  be  quickly  identified  and  easily  acted  upon.  In 
this  study  the  time-varying  effectiveness  criteria  were  presented  in  a 
numerical  matrix  display,  the  aid's  criteria  had  been  explained  to  all 
participants,  and  the  "good"  model  group  had  an  additional  one-page  summary. 
This  was  insufficient  for  purposes  of  the  task  presented  to  these  subjects  in 
the  time  available. 

What  would  be  desirable  in  an  actual  expert  system  would  be  an  explanation 
capability  that  operated  on  a  query-response  basis,  allowed  the  users  to  seek 
explanations  at  various  levels  of  detail,  and  made  it  easy  to  compare  their 
own  solutions  with  the  recommended  ones.  For  example,  during  their  problem 
solving  sessions,  participants  thinking  aloud  would  typically  make  remarks 
such  as:  "I  don't  understand  why  (Target  X)  has  such  a  high  priority;  I  would 
have  selected  (Target  Y) ; "  or  "I  would  always  hit  unsheltered  aircraft."  An 
expert  should  be  able  to  compare  the  factors  comprising  the  score  of  his  own 
choice  with  those  comprising  the  score  of  the  recommended  choice,  so  that  he 
can  determine  whether  or  not  to  accept  the  recommendation;  for  a 
less -than- expert,  the  comparison  would  be  beneficial  as  a  training  aid. 

In  more  general  terms  the  findings  of  this  study  suggest  that  a  self- learning 
system,  or  one  in  which  values  change  as  a  function  of  the  planning 
time-horizon,  should  have  certain  features  to  facilitate  both  explanation  and 
training.  These  features  are  described  below. 

5 . 1  Good  Conceptual  Model  of  the  Expert  System 


In  all  the  performance  comparisons,  participants  who  were  given  a  brief 
summary  of  the  aid's  sequential  processes  (i.e.,  who  had  a  relatively  good 
conceptual  model  of  the  aid)  did  better  than  those  who  were  not  given  this 
summary.  This  is  consistent  with  the  findings  of  Lehner  and  Zirk  (1987),  and 
reinforces  the  importance  of  ensuring  that  the  user  of  a  system  has  a  good 


mental  model  of  the  system,  regardless  of  his  model  of  the  problem  domain.  In 
the  present  study  this  system  model  was  presented  simply  in  the  form  of  a 
series  of  narrative  paragraphs.  In  other  contexts  (and  perhaps  even  in  the 
present  context)  other  representations  such  as  graphic  function  flow  diagrams 
or  network  diagrams  might  be  even  more  effective.  The  most  appropriate  format 
will  depend  on  the  characteristics  of  the  system  being  portrayed,  but  in 
general  a  graphic  representation  is  likely  to  be  more  effective  than  a  textual 
one. 

5.2  Display  Orientation 

There  was  no  evidence  that  user- oriented  displays  (maps  and  individual  target 
data  sheets)  resulted  in  either  better  or  worse  performance  than  aid-oriented 
displays  (data  matrices).  The  aid-oriented  displays  furnished  significantly 
more  information  in  a  single  table,  obviating  the  need  to  scroll  through  a 
series  of  tables  to  compare  the  characteristics  of  several  targets,  and 
therefore  should  have  been  preferable.  In  an  actual  system  there  would  be  no 
reason  not  to  have  both  types  of  format  available,  to  be  selected  at  the 
option  of  the  user. 

The  one  significant  difference  between  the  two  display  types  was  that  the 
aid- oriented  displays  apparently  encouraged  users  to  refer  more  frequently  to 
the  expert  assessment  tables,  which  contained  information  essential  to 
accurate  performance.  However  this  difference  did  not  result  in  better 
performance.  It  is  likely  that  better  performance  would  have  resulted  if  the 
explanatory  subsystem  had  been  more  effectively  designed  (see  below),  but  the 
results  offer  no  basis  for  preferring  one  type  of  format  over  the  other. 

5 . 3  Comparative  Method  of  Explanation 

As  noted  earlier,  the  participants  typically  worked  through  the  targeting 
problem  using  their  own  criteria  and  heuristics,  compared  their  results  with 
the  aid's  recommendations,  and  then  sought  an  explanation  for  the  differences. 
Although  much  of  the  needed  information  was  expressed  in  the  benefit/cost 
calculation  for  each  target  as  well  as  the  expert  assessment  tables  of  enemy 
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threat,  costs  of  friendly  attack,  and  effects,  these  tables  were  not  designed 
specifically  to  foster  direct  comparison  of  a  few  cases  presented  in  close 
proximity,  as  an  interactive  system  could  do  easily.  Displays  that  allowed 
users  to  select  two  or  more  cases  for  direct  comparison  at  various  levels  of 
detail  (see  below)  would  be  compatible  with  the  procedures  used  by  the 
participants  in  this  study. 

5.4  Hierarchical.  Structure 

Combined  with  the  comparative  method  of  explanation,  a  hierarchical  structure 
is  recommended  that  would  allow  the  user  to  compare  explanations  at  various 
levels  of  detail.  Thus,  at  the  highest  level  in  the  target  prioritization 
problem,  the  user  might  select  his  own  highest  priority  target  for  comparison 
with  the  aid's  top  one,  two  or  three  recommendations,  and  ask  to  see  simply 
the  benefit/cost  ratio  for  each.  He  might  then  call  up  a  listing  of  the 
next-level  factors  that  go  into  this  computation  (enemy  sorties  available, 
sorties  reduced,  and  cost),  for  a  rapid  comparison.  More  detail  could  be 
sought  by  examining  the  components  of  each  target,  the  sortie  reduction 
effects  and  the  costs  of  attacking  each  component.  Finally,  if  desired,  the 
user  could  call  up  the  equation  or  algorithm  used  by  the  aid  to  determine  its 
priorities. 

5 . 5  Time  Dependencies 

For  an  expert  system  that  adapts  to  a  changing  situation,  or  one  whose  values 
change  with  the  planning  time-horizon,  the  relevant  time -dependent  values  must 
be  made  very  evident  to  the  user.  In  this  study  the  value  of  attacking 
various  target  components  changed  as  the  planning  horizon  changed  from  three, 
to  two,  to  one  day  before  the  effect  was  desired.  The  Expert  Assessment  of 
Effects  table  showed  these  changing  values  for  each  type  of  target  component, 
but  the  participants  apparently  missed  or  disagreed  with  the  time -dependent 
effects,  or  simply  mis-read  the  table  despite  its  being  explained  to  them. 

For  the  problem  posed  in  this  study,  it  would  be  desirable  for  the  user  to 
enter  the  planning  day  of  interest  and  have  the  relevant  data  in  the  table 
highlighted  to  attract  the  user's  attention.  Similarly,  highlighting  could  be 
used  in  a  self-learning  system  to  call  a  user's  attention  to  data  or  values 
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that  have  changed  as  a  result  of  a  changing  situation.  In  the  latter  case  an 
alerting  message  would  be  appropriate  any  time  a  change  occurred  to  make  sure 
the  user  is  aware  of  a  change;  the  highlighting  would  remain  on  each  display 
with  relevant  data  until  the  user  has  called  it  up  once,  or  until  a  new  change 
has  occurred. 
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6.0  CONCLUSIONS 


On  the  basis  of  experimental  findings,  the  following  conclusions  are  derived: 

For  users  of  an  expert  system,  a  good  conceptual  model  of  the  system  is  more 
important  than  displays  that  are  consistent  with  the  users'  model  of  the 
problem  (geographical,  top-down  and  simplified).  Displays  that  are  relatively 
aid-oriented  (data- intensive  matrices)  are  just  as  effective,  and  tend  to 
encourage  users  to  refer  to  underlying  expert  assessments,  values  on  which  the 
aid's  algorithms  operate. 

Expert  system  users  may  apply  criteria  and  heuristics  that  differ  from  those 
of  the  experts  whose  knowledge  is  represented  in  the  system.  The  aid  should 
be  designed  to  allow  users  to  compare  their  solution  with  the  aid's 
recommendations,  to  facilitate  learning  as  well  as  to  provide  a  clear  basis 
for  experienced  users  to  disagree  with  the  aid's  recommendations  and 
substitute  their  own. 

The  aid's  explanation  subsystem  should  be  structured  hierarchically,  to  permit 
comparison  of  alternative  solutions  at  several  different  levels  of  detail. 

In  a  self- learning  system,  or  one  whose  values  change  with  the  situation  or 
with  the  planning  time-horizon,  it  is  important  to  alert  the  user  to  the  fact 
that  a  change  has  occurred  and  to  highlight  the  data  that  have  changed,  at  all 
levels  in  the  explanation  hierarchy. 
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APPENDIX  A 

INSTRUCTIONS  AND  GUIDANCE  GIVEN  TO  ALL  PARTICIPANTS 


A-l 


GENERAL  INSTRUCTIONS 


Introductions . 

We  represent  a  company,  Decision  Science  Consortium,  under  contract  with  the 
Armstrong  Aerospace  Medical  Research  Lab  at  Wright-Patterson  Air  Force  Base. 
The  purpose  of  our  research  contract  is  to  investigate  how  people  interact 
with  self-learning  expert  systems,  that  is,  decision  aiding  systems  that  may 
adapt  to  changes  in  an  evolving  situation.  We  are  not  promoting  the 
development  of  such  a  system.  But  there  is  a  lot  of  R&D  being  conducted  on 
various  types  of  decision  aids  and  our  concern  is  to  help  make  sure  that  if 
they  are  developed  for  operational  use,  they  are  designed  to  meet  the  needs  of 
their  users. 

I  want  to  assure  you  that  this  exercise  is  not  a  test  of  your  performance.  We 
want  to  get  some  information  about  your  background  to  help  us  interpret  the 
results,  but  the  data  that  we  get  during  the  exercise  will  be  treated  in  the 
aggregate,  and  no  names  will  be  used  in  our  report. 

We  have  chosen  the  tactical  air  strike  targeting  problem  as  the  context  for 
this  research- -again,  not  because  we  are  promoting  an  expert  system  for  this 
task  but  because  there  has  been  some  previous  work  on  a  target  prioritization 
aid  and  we  are  familiar  with  it.  In  addition,  Jack  is  a  retired  TAC  officer 
with  operational  experience  in  a  TACC,  and  he  can  answer  any  of  your  questions 
dealing  with  the  scenario  that  we  have  generated. 


For  purposes  of  this  exercise  we  will  ask  you  to  play  the  role  of  a  targeteer 
in  a  TAC  Headquarters  in  Central  Europe.  But  instead  of  generating  target 
recommendations ,  we  will  ask  you  to  assume  that  there  is  an  expert  system  that 
processes  available  data  and  presents  these  recommendations  to  you.  Your  task 
in  this  exercise  is  to  figure  out  why  the  system  generated  the  list  that  it 
did. 


We  will  set  the  stage  with  an  initial  scenario  that  is  fairly  typical  of 
tactical  air  operations  in  the  European  theater.  After  you  get  the  system's 
recommended  target  list,  you  will  have  several  data  displays  available  in  this 
notebook  to  help  you  figure  out  the  reasons  for  the  recommendations.  We  are 
interested  in  seeing  which  of  these  displays  you  use,  which  you  find  most 
helpful,  and  why.  If  you  can  think  aloud  while  you  are  solving  the  problem, 
that  would  be  most  helpful.  When  you  think  you  have  completed  the  task,  let 
us  know.  We  will  ask  you  a  few  more  questions  at  that  time. 

We  have  allowed  two  hours  for  the  exercise.  We  think  this  is  enough  time  for 
the  problem,  but  we  don't  want  you  to  rush  through  it. 

Finally,  we  would  appreciate  your  not  discussing  this  exercise  until  we  have 
completed  all  the  sessions.  Do  you  have  any  questions  about  the  procedure? 


l 
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COMMANDER'S  OBJECTIVE  STATEMENT 


The  CINCENT  objective  is  to  attain  localized  air  superiority  over 
the  area  of  Fulda  and  Hannover  in  three  days  so  that  on  "Day-4" 
maximum  emphasis  can  be  placed  on  Offensive  Air  Support.  COM- 
FOURATAF  is  requested  to  undertake  this  task. 

A.  In  response  to  Soviet-Warsaw  Pact  aggression,  which  includes 
incursions  into  NATO's  Central  Region  and  the  occupation  of 
West  Berlin,  the  Ministers  of  Defense  through  Hg  SHAPE  have 
granted  unrestricted  cross-border  authority  for  conventional 
offensive  operations  against  East  Germany,  Czechoslovakia, 
Poland,  and  the  Western  military  districts  of  the  USSR. 
Cross-border  activity  will  commence  on  "Day-1".  Attainment 
of  localized  Air  Superiority  over  Fulda  and  Hannover  and  the 
approaches  thereto  out  to  a  distance  of  150nm,  should  be 
completed  at  the  conclusion  of  "Day-3".  Consistent  with  the 
requirement  for  localized  air  superiority,  maximum  possible 
concentration  is  to  be  placed  on  attacks  against  Pact 
fighter-bomber/ground  attack  capable  aircraft  that  possess 
the  combat  radius  to  operate  with  these  areas.  Specifi¬ 
cally,  the  Flogger  D/J  and  the  Frogfoot.  At  the  conclusion 
of  "Day-3",  the  build-up  of  Central  Region  ground  forces 
should  be  sufficient  to  mount  a  counter-attack  which  is 
scheduled  to  commence  "Day-4". 

B.  To  insure  the  greatest  chance  for  success  of  this  undertak¬ 
ing,  CINCENT  has  also  directed  that  the  build-up  of  forces 
be  afforded  the  maximum  degree  of  protection  concurrently 
with  offensive  operations.  Accordingly,  all  air  defense- 
capable  aircraft  will  be  utilized  in  a  defensive  counterair 
role.  Apportionment  guidance  for  "Day-1"  through  "Day-3"  is 


offensive  counterair 
interdiction 
offensive  air  support 


45% 

10% 

45%. 


ADDITIONAL  INFORMATION 


For  purposes  of  this  experiment: 

You  will  consider  only  offensive  counterair  missions  against 
enemy  airbases. 

Only  16  enemy  airbases  are  considered. 

For  each  enemy  airbase  only  10  components  are  described  in 
the  Air  Installation  File. 

Only  nine  enemy  aircraft  are  listed  in  the  Air  Order  of 
Battle  files. 

The  target  prioritization  includes  enough  enemy  airbases  and 
components  to  require  more  than  300  F-16  sorties  to  carry 
out  the  attacks. 


Please  keep  in  mind: 

The  purpose  of  this  experiment  is  to  investigate  how  people 
would  interact  with  certain  kinds  of  systems.  Consequently, 
although  we  will  be  happy  to  answer  any  questions  you  may 
have  about  the  TPT,  there  may  be  some  questions  we  cannot 
answer  until  after  the  experiment  is  complete.  Also,  there 
may  be  times  when  you  feel  that  there  are  some  necessary 
displays  that  the  TPT  should  have,  but  doesn't.  Please  bear 
with  us,  this  too  is  part  of  the  experiment.  (At  the  end  we 
will  want  to  know  what  additional  displays  you  would  have 
liked  to  have.) 
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HOW  THE  TARGET  PRIORITIZATION  TOOL  WORKS 


The  TPT  assigns  a  target  value  (priority)  to  each  enemy  airbase 
and  recommends  components  to  attack  at  each  airbase.  The  TPT  as¬ 
signs  target  values  by  proceeding  through  a  series  of  calcula¬ 
tions  that  allow  it  to  estimate  the  benefit-to-cost  ratio  for  at¬ 
tacking  each  enemy  airbase  and  components  on  that  airbase.  The 
following  pages  describe  some  of  the  calculations  performed  by 
the  TPT. 

To  illustrate  these  calculation  assume  that  we  have  the  following 
situation: 

1.  The  only  aircraft  of  interest  is  the  Flogger  D/J.  Its  nor¬ 
mal  sortie  rate  is  3  for  minimum  distance  flights  of  200  or 
less.  Its  maximum  range  is  400  miles. 

2.  The  airbase  HOFT  has  10  Flogger  D/Js.  It  is  300  miles  from 
Area  of  Interest. 

3.  The  Air  Installation  File  indicates  the  following  about 
HOFT: 

jf  Unsheltered  A/C: 
t  A/C  Shelters: 

#  Revetments: 

Launch  &  Recovery  Surfaces: 

#  Command  HQs : 
v  Control  Sites: 

#  Munition  Dumps: 

#  POL  Sites: 

#  Maintenance  Facilities: 
i  Dumps  for  Spares: 


Typically  around  10 
20 
12 

10  aim  points 
2 

3 
6 

4 
4 
2 


NUMBER  OF  SORTIES  OF  INTEREST  AT  ENEMY  AIRBASE 


For  each  a/c  of  interest  calculate: 

(#A/C  OF  INTEREST) * (SORTIE  RATE) * (DISTANCE  REDUCTION  FACTOR) 
where  the  DISTANCE  REDUCTION  FACTOR  is  equal  to 
0  if  distance  to  AOI  >  MAX  DISTANCE, 

1  if  distance  to  AOI  <  MIN  DISTANCE, 
otherwise, 

DISTANCE  to  AOI  -  MIN  DISTANCE 
MAX  DISTANCE  -  MIN  DISTANCE. 


For  example,  for  HOFT  the  calculation  is 

r A/ C  OF  INTEREST  =  10 

SORTIE  RATE  =  3 

DISTANCE  REDUCTION  FACTOR  =  0.5 

#  SORTIES  OF  INTEREST  =  15 
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f  F-16S  REQUIRED  TO  ATTACK  ENEMY  AIR  BASE 


For  each  component  to  be  attacked  calculate 
(I  ATTACKS) 

*(#  F-16S  REQ'D  FOR  PROTOTYPE  BASE) 

* (ACTUAL  CAPACITY) /(PROTOTYPE  CAPACITY), 

and  sum  over  components. 


For  instance,  for  HOFT  assume  a  Plan  of  Attack  that  involves  two 
attacks  on  Revetments  and  one  on  Launch  and  Recovery  Surfaces: 


Revetments 


#  ATTACKS  2 

#  F-16  FOR  PROTOTYPE  10 
ACTUAL  CAPACITY  20 
PROTOTYPE  CAPACITY  10 


L&R  Surfaces: 
1 
17 
12 
12 


a  F-16S  REQ'D 


40  +  17 


57 
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EXPECTED  NUMBER  OF  ENEMY  SORTIES  REDUCED 


For  each  attack  on  each  component  look  up  (in  Effects  Knowledge 
Base)  the  estimated  MAXIMUM  REDUCTION  and  %  REDUCTION  in  sortie 
rate.  For  each  attack  on  each  component,  the  REMAINING 
CAPABILITY  is  calculated  as  follows: 

MAX  REDUCTION  %  REDUCTION 

REMAINING  CAPABILITY  =  1  -  -  *  - 

100  100 


The  EXPECTED  NUMBER  OF  SORTIES  REMAINING  is 

NUMBER  OF  SORTIES  OF  INTEREST 

*  INITIAL  CAPACITY/100 

*  REMAINING  CAPABILITY  AFTER  1st  COMPONENT  ATTACK 

*  REMAINING  CAPABILITY  AFTER  2nd  COMPONENT  ATTACK 


*  REMAINING  CAPABILITY  AFTER  Mth  COMPONENT  ATTACK 


Finally  the  EXPECTED  NUMBER  OF  SORTIES  REDUCED  is  equal  to 

NUMBER  OF  SORTIES  OF  INTEREST 
-  EXPECTED  NUMBER  OF  SORTIES  REMAINING. 


For  instance  for  HOFT,  for  two  attacks  against  Revetments  and  one 
against  Launch  and  Recovery  Surfaces  we  may  get 

Attack  1  Attack  2  Attack  3 

Component  Revetment  Revetment  L&R  Surfaces 


MAX  REDUCTION 

25 

25 

30 

%  REDUCTION 

40 

40 

50 

10 

10 

15 

#  SORTIES  OF  INTEREST 

22 

15 

INITIAL  CAPACITY 

a 

100 

REMAINING  CAPABILITY 

1ST  ATTACK 

a 

.9 

REMAINING  CAPABILITY 

2ND  ATTACK 

a 

.9 

REMAINING  CAPABILITY 

3RD  ATTACK 

= 

.85 

EXPECTED  #  SORTIES  REMAINING 

10.3 

EXPECTED  NUMBER  SORTIES  REDUCED  =  15  -  10.3 

=  4.7 


TARGET  PRIORITY 


The  TARGET  PRIORITY  of  an  airbase  is  determined  by  calculating 
the  BENEFIT-TO-COST  RATIO  for  attacking  the  components  selected 
in  the  Plan  of  Attack. 

The  BENEFIT-TO-COST  RATIO  is  equal  to 

EXPECTED  NUMBER  OF  SORTIES  REDUCED 


(#  F-16s  REQ'D  +  #  F-lllS  REQ'D  +  l)/2 

For  instance,  for  HOFT  for  two  attacks  against  Revetments  and  one 
against  Launch  &  Recovery  Surfaces  we  get: 

EXPECTED  #  OF  SORTIES  REDUCED  =  4.7 

NUMBER  OF  F-16s  REQUIRED  =  57 

NUMBER  OF  F-llls  REQUIRED  =  28 


BENEFIT-TO-COST  RATIO  =  4 . 7/ [ (57+28+1) /2 ] 

=  .11 


The  TPT  determines  the  high  priority  airbases  by  selecting  the 
airbases  and  airbase  components  that  maximize  BENEFIT-TO-COST 
RATIOS.  HOFT  has  a  very  low  BENEFIT-TO-COST  RATIO.  It  would 
have  a  very  low  TARGET  PRIORITY. 
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WEATHER  CONDITIONS 
EFFECT  WINDOW 

ENEMY  AIRCRAFT  OF  INTEREST 
FRIENDLY  SORTIES  PER  DAY 


BAD  WEATHER  IS  DEFINED  AS  SITUATIONS  IN  WHICH 
ONE  OR  BOTH  OF  THE  FOLLOWING  CONDITIONS  ARE  MET: 


-  LESS  THAN  3  MILES  VISIBILITY 

-  LESS  THAN  3000  FOOT  CEILING 


SPECIFY  THE  DAYS  ON  WHICH  BAD  WEATHER  IS  PREDICTED: 

0)  TODAY 

1)  DAY  1 

2)  DAY  2 

4)  DAY  4 

5)  DAY  5 

6)  DAY  6 

7)  DAY  7 

LIST  THE  DESIRED  DAYS:  N/A 
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SPECIFY  THE  EFFECT  WINDOW 

HOW  MANY  DAYS  IN  THE  FUTURE  DO  YOU  WANT  THE  EFFECT 
TO  START  (FIRST  DAY  OF  PLAN  IS  DAY  1)?  4 

HOW  MANY  DAYS  DO  YOU  WANT  THE  EFFECT  TO  LAST? 

(MAX  =  7):  1 
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SPECIFY  AIRCRAFT  OF  INTEREST 


1) 

FENCER  (FNCR) 

6) 

FLANKER  (FLNKR) 

2) 

FLOGGER  D/J  (FLGR 

DJ) 

7) 

FISHBED  H  (FSBD  H) 

3) 

FLOGGER  G/K  (FLGR 

GK) 

8) 

FOXBAT  S  (FXBT  S) 

4) 

FROGFOOT  (FRGFT) 

9) 

FOXBAT  B/D  (FXBT  BD) 

5) 

FULCRUM  (FLCRM) 

LIST  AIRCRAFT  TYPES  OF  INTEREST  (BY  NUMBER):  2,4 
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SUBJECT  INFORMATION  SHEET 


Page  1 


NAME:  _ _ 

RANK:  _ 

CURRENT  ASSIGNMENT:  _ 

YEARS  OF  EXPERIENCE  a*  TARGETEER: 
TIME  IN  CENTRAL  EUROPE: 


For  each  high  priority  target  on  each  day  of  the  attack: 


Page  2 


(1)  Indicate  by  number  those  factors  (from  the  List  of  Factors)  that  you 
believe  contributed  significantly  to  its  high  value  as  a  target. 


(2)  Rank  the  components  selected  for  attack  at  that  airbase  according  to 
their  relative  contribution  to  the  assigned  value  for  that  target. 
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First  Priority  Target:  Mtmon 

Factors:  _  _ _ 


Component  Rank  Order: 


Second  Priority  Target:  Altenburg 

Factors:  _  _  _ 


Component  Rank  Order: 


Third  Priority  Target:  Trollenhagen 

Factors:  _  _  _ 


Component  Rank  Order 


DAY  2 


First  Priority  Target:  Trollenhagen 

Factors:  _  _  _ 


Second  Priority  Target:  Mimon 

Factors:  _  _ 


Component  Rank  Order: 


Third  Priority  Target:  Grossenhain 

Factors:  _  _  _ 


Component  Rank  Order: 
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DAY  3 
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I 


First  Priority  Target:  Altenburg 

Factors:  _ _  _  _ 


Component  Rank  Order: 


Second  Priority  Target:  Mimon 

Factors :  _ _  _ 


Component  Rank  Order: 


Third  Priority  Target:  Trollenhagen 

Factors:  _  _  _ 


Component  Rank  Order: 
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GENERAL  QUESTIONS 


Page  6 


Which  displays  helped  you  bosC  in  understanding  the  system? 
For  each  one,  why? 


If  you  could  design  the  contents  and  formats  of  displays  that  would 
be  "ideal"  for  you,  what  would  they  look  like?  (Use  the  blank 
sheets  available.) 


For  each  one,  why?  (Describe  below  or  on  corresponding  sheets.) 


GENERAL  QUESTIONS 
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1.  How  do  you  define  the  targeting  problem? 
How  would  you  go  about  solving  It? 


2.  How  do  you  think  the  aid  defines  and  solves  the  targeting  problem? 


3.  How  do  you  think  an  ideal  aid  could  support  you  in  solving  the  targeting 
problem? 
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»  AffHOX.  to  ^ 

trttrt 


or 
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AIRBASE:  MIMON 


Note:  This  is  a  sample.  A  similar  table 
was  available  for  each  target. 


BASIC  ENCYCLOPEDIA  NO 
MAP  coordinates: 
DISTANCE  TO  AOi: 

#soi: 


0103-6561129 

50  37  12  n,  14  43 

175 

98 


AIR  ORDER  OF  BATTLE 

FENCER  (FNCR):  _ 

FLOGGER  D/J  (FLGR  Dj) :  _ 

FLOGGER  G/K  (FLGR  GK> :  _ 

FROGFOOT  (FRGFT)  :  _ 20. 

FULCRUM  (flcrm):  _ 15. 

FLANKER  (FLNKR) : 

FISHBED  H  (FSBD  H) : 

FOXBAT  S  (FXBT  S)  *. 

FOXBAT  S/D  (FSBT  BD> : 

AIR  INSTALLATION  INFORMATION 
AIRCRAFT  IN  OPEN  (ACO) : 

AIRCRAFT  IN  SHELTERS  (ACS) : 

AIRCRAFT  IN  REVETMENTS  <ACR> : 
MAINTENANCE  FACILITIES  (MXF)  .* 

L  &  R  SURFACES  (L&R) : 

POL: 

MUNITIONS  DUMPS  (MUN) : 

COMMAND  HEADQUARTERS  (CMD) : 
CONTROL/RADAR  FACILITIES  <CNT> : 

spares: 


20 


8 


9 

2 

3 

3 


(aircraft) 

(shelters) 

(revetments) 

(facilities) 

(aimpoints) 

(aimpoints) 


48  e 
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APPENDIX  C 

AID-ORIENTED  DISPLAYS 


c-i 


C 


PLAN  OF  ATTACK,  DAY 


DAMGARTEN 
TROLLEN  HAGAN 
GROSSEN  HAIN 

MIMON _ 

STRACHOWICE 


PLAN  OF  ATTACK,  DAY  2 


DAMGARTEN 
TROLLEN  HAGAN 
GROSSEN  HAIN 

MIMON _ 

STRACHOWICE 


PLAN  OF  ATTACK.  DAY  3 
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APPENDIX  D 
CORRECT  RESPONSES 
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LIST  OF  FACTORS 


1.  Relatively  short  distance  between  airbase  and  Area  of  Interest 

2.  Relatively  high  capacity  of  components  selected  for  attack  (e.g.,  rela 
tively  more  shelters). 

3.  Airbase  contains  relatively  more  enemy  a/c  of  interest. 

4.  Relatively  low  capacity  of  components  selected  for  attack  (e.g.,  rela¬ 
tively  fewer  shelters). 

5.  Relatively  low  number  of  friendly  sorties  required  to  attack  selected 
components . 

6.  Relatively  high  sortie  rate  of  a/c  of  interest  at  enemy  airbase. 

7 .  Relatively  high  %  reduction  in  sortie  rate  for  attacking  selected  com¬ 
ponents  . 


1.  Yes  if  distance  less  than  200 
No  if  distance  greater  than  200 
?  if  distance  equals  200 


2 .  No  always 


3.  Yes  if  45  a/c  of  interest 

No  if  15  or  0  a/c  of  interest 
?  if  30  a/c  of  interest 


4.  Yes  if  capacity  less  than  average  of  the  16  airbases 
No  if  capacity  more  than  average  of  the  16  airbases 
?  if  capacity  equals  the  average  of  the  16  airbases 


5.  Discarded — redundant  with  #4. 


6.  Yes  if  airbase  contains  only  Frogfoots 

No  if  airbase  contains  only  Flogger  D/J's 
?  if  airbase  contains  both 

7 .  No  always 
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3. 


L&R  SFCS 


6. 


Second  Priority  Target:  Altenburg 

Factors:  Yes  No  ?  ? 


Yes 


No 


Component  Rank  Order: 

1.  MAIN  FC _ 

2.  RVTMNTS _ 

3.  SPARES 


4.  L&R  SFCS 

5.  _ 

6. 


Third  Priority  Target: 
Factors:  ? 


Trollenhagen 
No  Yes  Yes 


-Xes... 


Component  Rank  Order: 

1.  MAIN  FC _ 

2-  —SPARES _ 

3.  RVTMNTS _ 


4. 

5. 

6. 
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DAY  2 


First  Priority  Target: 


Factors 


Component  Rank  Order: 
1.  RVTMNTS  ______ 


UNS  A/C 


MAIN  FC 


Second  Priority  Target: 
Factors:  Yes 


Component  Rank  Order: 
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